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ABSTRACT 



The major hypothesis was that students whose 
teachers received training in the use of media would attain higher 
mean achievement scores than students whose teach, rs received no 
media training. Testing of third -grade students in reading, language 
arts, arithmetic, and intelligence did not support the hypothesis. 
However, teachers who had been trained in the use of media did use 
them pore and were more favorable toward their use. The utilization 
and attitude findings were determined by questionnaires and 
observations. The results of the study are judged inconclusive, and 
questions are raised about the correlation between total learning and 
that learning sampled by standard ized achievement tests. (JK) 



&\oog 8(0 



PJ 

iC\ 



oo 

- 4 - 



Lsj 



U,&0€FAttTMENTOF HEALTH, EDUCATION 
SWEtFARG 
OFFICE OF EDUCATION 
THIS DOCUMENT HA& BEEN REPRODUCED 
EXACTLY AS RECEIVED FROM THE PERSON Ofl 
ORGANIZATION ORIGINATING IT. POINTS OF 
VIEW OR OPINIONS STATED DO NOT NECES- 
SAftrLY REPRESENT OFFICIAL OFFICE OF COO 1 
CATION POSITION On POLICY 



EVALUATION REPORT 
MEDIA UTILIZATION PROJECT 



A Joint Project of the Division of 
Educational Media and the Comprehensive 
School Improvement Project 



North Carolina State Department of Public Instruction 

March 30,. 1971 




1 



_ 1 _ 



Participating Schools 
With Number of Classes 



Experimental 



Control 



Number of 



School Classes 

Washington Street Elementary 4 

Plymouth, North Carolina 

Hampton Elementary 3 

Greensboro, North Carolina 

Elon College Elementary 3 

Elon College, North Carolina 

Weaverville Primary 4 

Weaverville, North Carolina 

Drexel- Primary 3 

Drexel, North Carolina 



Number of 



School Classes 

Reeds Elementary 3 

hexing ton, North Carolina 

North Rowan Primary 4 

Spencer, North Carolina 

Ahemethy 3 

Gastonia, North Carolina 

H. B, Sugg 3 

Faiwille, North Carolina 

Rockford Street 3 

Mt. Airy, North Carolina 
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Background and Analysis 

In the summer of 1969, the Comprehensive School Improvement Project (CSIP) and 
the Division of Educational Media began planning a cooperative project designed 
to test pupil cognitive learning resulting from specialised training in the use 
of audiovisual techniques and equipment. The rationale for this project implied 
that, as a result of training in the utilisation of media, teachers would become 
more effective in using equipment presently available in their school situations. 

CSIP schools were deemed appropriate for this project since they were currently 
involved with various approaches to team teaching. In addition, previous sur- 
veys had shown that utilisation of audiovisual equipment formed a significant 
portion of the duties of the teacher aide. (Aides are provided for each CSIP 
team from CSIP funds.) 

In planning for this project, it was assumed that a "typical" school does not 
obtain mavi mum effectiveness from its media equipment. The training was designed 
to increase media utilisation which would result in increased learning on the 
part of students. It was anticipated that project schools would increase their 
holdings of media materials , but not equipment, as one result of this experiment. 
Given these considerations, it was expected that the actual training would vary, 
depending upon expressed teacher needs, as well as upon each school's media 
situation. 

Although media utilisation was the focus of this experiment, the outcome of in- 
creased usage was not considered to be a sufficient measure of the success of the 
endeavor. Instead, the outcomes in terms of student learning were selected as 
being more significant. Thus, the major hypothesis for this study was that 
students whose teachers received training in the utilisation of media would attain 
higher mean achievement scores than control students. Two secondary hypotheses 
were tested. It was hypothesised that, as a result of the training, experimental 
teachers would: 1) increase their effective usage of audiovisual equipment and 

materials, and 2) demonstrate an improvement in attitude toward the use of audio- 
visual techniques. An additional consideration was an evaluation of the training 
procedures. 

Prior to the opening of school in September 1969, ten CSIP schools with third- 
grade teams were chosen as possible participants. All ten readily agreed to 
cooperate. In October 1969, all third-grade pupils were tested with the Science 
Research Associate's Primary Mental Abilities Test . Schools were assigned to 
pairs matched on the basis of these intelligence scores. In none of the five 
pairings were there any significant differences as measured by the IMA test 
results. One member of each pair of schools was randomly selected to receive 
the media training, while the matched school provided the control component. 

Prior to beginning the training, all students were tested for achievement by 
Form D of the Science Research Associates ' Achievement Test . This yielded a set 
of pretest scores. Immediately following this testing, the training sessions 
were begun. Each experimental school received a minimum of three visits from a 
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consultant in the Division of Educational Media.. Although original plans called 
for the training to he completed by February 1971, personnel limitations and 
scheduling difficulties made it necessary to extend the training throughout the 
duration of the project. Each inservice session was approximately one-half day 
in length and was conducted during the working day at the school. Among the 
topics considered during these training sessions were; transparency production, 
use of the sixteen millimeter movie projector, use of filmstrips, and the 
application of audiotape directions to individualised instruction. At the first 
session the use of available local resources wets considered. The subject-matter 
orientation of the training sessions was secondary to utilization techniques, 
i ,e. the examples were not directed toward previously selected third-grade 
subjects . 

Following the training, both the experimental and control students were tested 
with Form C of the SRA Achievement Test in May 1970. A+ ^e same time, all 
teachers (and principals) who attended the training ? jsions responded to a 
questionnaire designed to assess the secondary outcomes of the project. After 
the final testing phase of the project was begun, teachers of the control groups 
were informed of their’ role in this project and were brought to Raleigh for a 
one -day workshop during which they were given training somewhat comparable to 
that provided the experimental group during the year. 

The experimental design for testing the major hypothesis (of student achievement) 
was a modification of the prates t-posttest control group design. As mentioned 
earlier, schools were matched on the basis of intelligence scores. Since it was 
not possible to randomly assign students to the experimental and control groups, 
analysis of covariance was used in order that initial differences on the achieve- 
ment test could also be considered. For the analysis of covariance, all scores 
were entered into the Manova Program of the UNC Computer Library. The composite 
posttest score from Form C of the SRA. Achievement Test was used as a single 
criterion variable. The three major subscores of the pretest (Form D) were 
treated as covariates* Although schools had been matched on intelligence, the 
scores from the IMA test were also entered as a covariate. Thus, the analysis 
considered four covariates: reading, language arts, arithmetic, and intelligence, 

and a single criterion. The two secondary hypotheses (increased utilisation and 
attitude) were evaluated by questionnaire results and observations. 

Complete sets of scores were obtained for 775 students - 440 experimental and 
335 control. The means and standard deviations for this group are presented in 
Table I. The experimental group's mean on the posttest (criterion) exceeded the 
mean for the control group by 3. 04. However, when initial achievement 
differences were taken into account by analysis of covariance, the control group's 
mean exceeded that of the experimental group by 3.83 (see adjusted criterion in 
Table 1). These differences, though small, are shown to he statistically 
significant by the results presented in Table II (F Test for Treatment). The 
ultimate difference favored the control group by 3.83 raw score points. Thus, 
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the covariance analysis indicated that i£ the groups had been originally equal 
(equal on the achievement as well as the intelligence pretest) the control group 
would have exceeded the experimental by almost four points. 

The second F Test in Table II was also significant. This test for regression 
indicated that the original differences in achievement were more than would be 
expected by random assignment. The regression coefficients presented in Table 
IB! provide an approximation of the effects of each of the separate covariates. 
These coefficients indicate that the pretest reading and math scores were 
responsible for most of the differences. The low (.08) coefficient for intelli- 
gence was to be expected since the groups were originally matched on intelligence. 

Table IV contains the direction and amount of adjustment made on the posttest 
score by the combination of all covariates. It is notable that the majority of 
the adjustments for the control schools were positive, while adjustments for the 
experimental schools were generally negative. The average adjustment for all ten 
schools was 17.5 raw score points. 

Given these results, the major hypothesis of student achievement gains could not 
be supported. Statistically, the results indicate that the training had a 
slightly negative effect on achievement as measured by the SRA Achievement Test . 
From a practical standpoint, these results could be judged inconclusive. Four 
raw score points out of two hundred is a very small difference. Most test 
conversion tables are not sensitive to differences of this degree. A comparison 
of these differences with the average adjustment of 17.5 points indicates that a 
replication of this project with a different assignment of schools to treatments 
might well yield insignificant results. 

Selected portions of the questionnaire data are presented in Tables V, VI and 
VH. Tables V and VI reveal generally positive attitudes toward the use of media. 
Three-fourths of the respondents indicated that their schools had increased 
utilization of media, and almost as many reported changed teaching practices as 
a result of the project. All respondents recognised that media can be of con- 
siderable help in the individualization of instruction, and ninety percent judged 
the demonstrations applicable for their classroom situations. These tables can 
be summarized as generally supportive for the secondary hypotheses of increased 
utilization of media and improved attitude toward use of media. 

The responses directly related to evaluation of the training procedures (Table VI) 
were not as consistent as the responses noted above. Seventy-five percent 
indicated a need for more training time. Thirty-five percent indicated a need for 
more materials, while another thirty-five percent found no difficulty in obtai ning 
materials. Perhaps indicative of the tone of these responses (except for the 
item concerning the amount of time) is that sixty-five percent felt better in- 
formed about newer technological developments. 
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Conclusions and Recommendations 

In the opinion of the writer, the results from this study are best judged incon- 
clusive. The slightly positive questionnaire responses fail to offset the 
negative results in the major analysis. Further consideration of the relation- 
ship between media training and utilization would be more productive if more were 
known about the correlation between total learning and that learning sampled by 
standardized achievement tests. The appearance of the reading score as the single 
most important covariate raises a question concerning the applicability of the 
achievement criterion which was employed. Some of the activities suggested in 
the training sessions were employed as a substitute for reading for information. 
This is not a suggestion that achievement is not the proper criterion - it is. 

The question being raised is rather "How much of the achievement measured by our 
instruments is separable from reading?". 

The results of the project indicated a definite need for more training time. In 
addition, it would appear that the project could have been improved by relating 
the training more directly to the achievement of students. Designers of future 
studies in the area of media might wish to consider developing achievement- 
oriented curriculum as an integral part of the training. Since many librarians 
and media specialists in the schools are highly competent, more positive results 
might have been yielded had they been assigned some of the responsibility for 
the training. 

The project could have progressed more rapidly and fully had better scheduling 
been implemented and more manpower allotted to the time-consuming task of test 
scoring. This would have increased the amount of time between the training of 
teachers and the final testing of students. 

The foregoing comments are not meant to be suggestive that this project was without 
value. The project yielded positive gains in attitude and utilization as measured 
by the questionnaire responses. It is unfortunate that time limitations precluded 
assessment of any possible long-range outcomes. 
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Table I 



Means and Standard Deviations for Criterion and Covariates 





COVARIATES 


criter: 


EON 


Number 


I.Q. 


Reading 


Lancuase Arts 


Mflth 


Post Test 


Adjusted 












Post Test 


Mean 


104.41 


42.77 


70.25 


46.65 


192.82 


189.87 


Experimental 140 














Standard Deviation 


26.11 


14.82 


20.28 


17.90 


52.40 




- * Mean 


106.84 


40.39 


68.06 ! 


44.40 


189.78 


193.70 


Control 335 














Standard Deviation 


25.49 


13.01 


17.28 


16.37 


49.01 
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Table II 



Analysis of Covariance for Experimental and Control Schools 



Source of Degrees of Sum of 

Variation Freedom Squares 



Mean F 

Square 



Treatment 

Regression 

Within 



1 

4 

769 



2637.0 

1745751.0 

261802.0 



2637.0 

436437.8 

340.5 



7.75* 
1281. 96^ 



•^Significant at .01 Level 



Table III 



Regression Coefficients for Covariates 



I.Q. 
Reading 
Language Arts 
Math 



.08 

1.41 

.62 

1.02 
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Table IV 



Comparison of Covariance Adjustments Between 
Experimental and Control Schools 



E x p e r i n 


i e n t a 1 1 


1 C o n t r c 


>1 


School 


N 


Rost test 


Adjusted* 
Post test 


Adjustment 


1 School 


N 


Post test 


Adjusted* 
Post test 


Adjustment 


11 


91 


220.9 


195.8 


- 


25.1 


86 


84 


222.5 1 


202.1 


- 20.4 


12 


78 


226.7 


190.6 


- 


36.1 


80 


74 


183.7 


183.9 


-r 0.2 


- 01 


84 


191.2 


180.4 


- 


H.l 


29 


74 


182.5 


188.3 


+ 5.8 


94 


89 


185.2 


188.2 




3.0 


36 


62 


168.6 


191.4 


+ 22.8 


41 


76 


168.6 


191.4 




22.8 ! 


I 74 


63 


173.2 


200.7 


+ 27.5 



*See Table III for Regression coefficients of the four covariates 
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Table V 

Selected Responses Indicating Attitude 
Toward Use of Audiovisual Materials 



Please check the word which most accurately reflects your use of audiovisual 
materials • 

Prior to Training Since the Training 



Considerable 


79$ 


Considerable 


89$ 


Some 


16$ 


Some 


11$ 


limited 


5$ 


limited 




None 




None 





Has your school increased its usage of audiovisual materials? (from outside 
sources) 

Yes 7 5$ 

No 25$ 

Does your school now have available, as a result of this training, more software 
that is usable in direct classroom teaching? 

Yes 67 % 

No 33$ 

Have you changed any of your teaching practices (methods) as a result of this 
training? 

Yes 71$ 

No 29$ 

Have you used any of the techniques recommended by representatives of the State 
Department for evaluating pupil progress? 

Yes 70$ 

No 30$ 

Was this a new practice for you? 

Yes 50$ 

No 8$ 

No Response 42$ 
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Table VT 



Selected Responses Indicating Attitude 
Toward Use of Media 



Ih your opinion, how much help can the new educational media provide for a 
primary school focusing on individualized instruction: 



How would you rate your 
and techniques? 



Considerable help 100$ 
Some help 0$ 
Limited help 0$ 
No help at all 0$ 

students' response to (your) 

Excellent 47$ 
Average 53 % 
Below average 0$ 
Undecided 0$ 



use of audiovisual materials 



Have you, as a result of this training, been able to identify new school or 
classroom needs for audiovisual equipment or materials? 



Yes 74 % 
No 26 % 



Were most of the materials used for demonstration by State Department personnel 
suited to your third-grade classroom situation? 

Yes 90$ 

No 10$ 

Do you think that mailed instructions could have increased the effectiveness of 
this training? 



Yes 24$ 
No 38$ 
Undecided . 38$ 
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Table VH 



Percent 

Responses 

25% 

75% 

35% 

35% 

100 % 

0 % 

90% 

05% 

60% 

0 % 

65% 

20 % 

15% 

50% 



Responses to Prepared Statements 
Describing the Media Utilization Project* 



The amount of training time was adequate 

There was not enough contact time with personnel from the 
State Department of Public Instruction 

There was no particular difficulty in obtaining satisfactory 
materials for classroom use 

The limited amount of materials, furnished from State sources, 
handicapped this training 

The teacher was allowed "freedom'’ to plan her own' program of media 
utilization 

Too much responsibility was placed on the individual teacher for 
media utilization 

Most of the techniques demonstrated during this training could 
be applied to third-grade classes 

The application of this training to actual classroom situations 
was not clear 

The State Department consultant was effective in communicating 
with the participants 

The State Department consultant did not seem to understand 
problems that actually exist in the field 

I am now better informed about new technological developments in 
audiovisual education 

There was very little new or unknown in any of these presentations 

The times selected for the training sessions were inconvenient 

The training times selected fitted in well with the school day 



^Percentage totals will not add to 100% since more than one response was allowed 



